Core experimental practice for machine learning

Caveats: There are ways to do model selection on the training set (using marginal likelihoods, or learning theory bounds), but these should be carefully justified. Sometimes, after making all model choices, you might fold the validation set into the training set, and retrain your best system before final testing. But by default you should not train on the validation set, so you can still use it to compare choices. Training or selecting choices on the test set is never ok.